Cross-Category Highlight Detection via Feature Decomposition and Modality Alignment

Learning an autonomous highlight video detector with good transferability across categories, called Cross-Category Video Highlight Detection(CC-VHD), is crucial for the practical application on video-based media platforms. To tackle this problem, we first propose a framework that treats CC-VHD as learning category-independent feature representation. Under framework, novel module, named Multi-task Feature Decomposition Branch which jointly conducts label prediction, cyclic reconstruction, and adversarial reconstruction to decompose features into two independent components: highlight-related component category-related component. Besides, align visual audio modalities one aligned space before conducting modality fusion, has not been considered in previous works. Finally, extensive experimental results three challenging public benchmarks validate efficacy of our paradigm superiority over existing state-of-the-art approaches detection.